Skip to main content

Incident Report: ETH/USD Avalanche(Self-funded) is not being updated

Date: 2023-12-1/2
Time: 22:30 (GMT+3)
Duration: 30 minutes

Description​

On Centurion Redstone api, Nodary api and Mertcan api detected deviation on ETH/USD Avalanche(Self-funded) which is not being updated

Root Cause​

The reason was it is not funded enough, but the calculation done wrong because of misinformation

Impact​

Had no impacts

Timeline​

  • 22:30 - Arda noticed the issue and started to investigate
  • 22:45 - Escalated through slack, #monitoring-team
  • 22:49 - Warren interacted
  • 23:01 - Bedirhan explained the issue and resolved confusion
  • 23:09 - Ali interacted
  • 23:19 - Bedirhan gave further information
  • 03:31 - Andrew pointed out the same issue
  • 23:01 - Burak interacted
  • 04:39 - Ashar interacted

Lessons Learned​

Even the previous transaction fees are roughly X, and current balance of the wallet is a number above X, Airseekers are trying to update with predefined Y gas unit. Therefore, to calculate it correct, one need to find fulfillment gas limit used in Airseeker which is shown in previous transactions "Gas Limit" section. Then find out the current gas price of the chain and multiply both numbers, and compare it with the current balance of the wallet.

Furthermore, "Self-funded Feeds to Monitor" section in Opsgenie Alerts Glossary needs to be edited and clarify which networks are being used.

Actions Taken​

N/A

Incident Reviewer(s)​

  • Arda
  • Warren
  • Bedirhan
  • Ali
  • Andrew
  • Burak
  • Ashar
Last updated on by Warren Anderson